<html>
<head><meta charset="utf-8"><title>extern languages? · t-cargo · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/index.html">t-cargo</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html">extern languages?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="239480117"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239480117" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joe Hendrix <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239480117">(May 19 2021 at 19:10)</a>:</h4>
<p>Hi, I'm fairly new to Rust, but interested in trying to support cross-language developer package management tools.  </p>
<p>I'm personally interested in this for building package management for <a href="https://github.com/leanprover/lean4/">Lean</a>, but would like to support a  variety of other languages.   I saw the work in the CXX package, and that looks really great, but was hoping for a solution that didn't require a custom <code>build.rs</code> file.</p>
<p>I have studied the Cargo implementation a bit and still have a lot to learn, but think support for additional languages could be done without breaking compatibility with existing Rust packages.  I should have a writeup of the approach available in a few days, but it basically involves introducing "external languages" similar to but orthogonal to the existing support for "external commands".  Each package that used an external language would need to identify the language in <code>Cargo.toml</code>.  For example,</p>
<div class="codehilite"><pre><span></span><code>cargo-features = [&quot;extern-language&quot;]

[package]
name = &quot;mypackage&quot;
version = &quot;0.0.1&quot;
language = &quot;lean&quot;
</code></pre></div>
<p>With this feature, Cargo  would invoke  an external command (e.g., <code>cargo-lang-lean</code>) for all language specific functionality.  At a minimum this would replace the call to <code>rustc</code> for compiling a package, but it would also include things like computing output files, fingerprints, and building documentation.  Cargo would still be responsible for building Rust packages and all functionality not specific to a particular package though.</p>
<p>Thoughts?  I recognize you have a small number of volunteer developers with limited time, and this could be a big ask.  I've started just a little bit down this path, and thought I'd mention the idea early to see if this could be a feature to eventually consider for upstream integration instead of doing something like a language-specific fork of cargo.</p>



<a name="239483473"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239483473" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239483473">(May 19 2021 at 19:34)</a>:</h4>
<p>I think that's likely to be a <em>massive</em> change, and one that would make Cargo substantially more complex. It would also need a definition for what precisely Cargo expects the command to <em>do</em> and to <em>output</em>. It's also likely to not fit what many people want from integration, because many people don't want cargo "driving", they want something else driving Cargo.</p>



<a name="239483580"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239483580" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239483580">(May 19 2021 at 19:35)</a>:</h4>
<p>However, there's an alternative that I think would potentially work for you, that we could use some help implementing if it interests you.</p>



<a name="239483930"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239483930" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239483930">(May 19 2021 at 19:38)</a>:</h4>
<p>There's a proposal for "artifact dependencies", which means you could declare a dependency on a cdylib or binary or similar.</p>



<a name="239483960"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239483960" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239483960">(May 19 2021 at 19:38)</a>:</h4>
<p>And there's a second proposal (not yet accepted) to allow <a href="http://build.rs">build.rs</a> scripts to build and provide such dependencies.</p>



<a name="239484022"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239484022" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239484022">(May 19 2021 at 19:38)</a>:</h4>
<p>The combination of those things would mean that you could declare a dependency on <code>mypackage</code>, and <code>mypackage</code> could then build Lean code and supply the resulting library, and Cargo would pull in that library and let you use it.</p>



<a name="239484041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239484041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239484041">(May 19 2021 at 19:39)</a>:</h4>
<p>Does that sound like something that might work for you?</p>



<a name="239485354"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239485354" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joe Hendrix <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239485354">(May 19 2021 at 19:48)</a>:</h4>
<p>There would need to be a bidirectional dependency.  It's more common that I would need a Lean package to depend on a Rust package, but also without something closer to first class support this wouldn't reduce the work required to maintain a separate package management system.  The current one in Lean is called <code>leanpkg</code>.  </p>
<p>Since others are wanting to have something else drive Cargo, could you share what other tools people are wanting to use?  Is this other build systems like Bazel or something else?  What I really want is a developer-oriented cross-platform package management system with easy per-package language-specific build integration.</p>



<a name="239487299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239487299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239487299">(May 19 2021 at 20:01)</a>:</h4>
<p><span class="user-mention" data-user-id="399826">@Joe Hendrix</span> I'm currently working on such a project myself, but in the interim, I don't think Cargo has a desire to become that system.</p>



<a name="239487577"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239487577" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239487577">(May 19 2021 at 20:03)</a>:</h4>
<p>I think we specifically want to see people using Cargo to build Rust, and using other things to build non-Rust, and connecting the two either by having Cargo drive some other system or having some other system drive Cargo. But either way, I don't think we want to have a concept of "foreign crates" integrated by third-party tools; the idea of <code>cargo-lang-lean</code> or similar as you described it would be a very "wide" interface that would also potentially need to update when Cargo updates.</p>



<a name="239487596"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239487596" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239487596">(May 19 2021 at 20:03)</a>:</h4>
<p>I don't see how we'd be able to commit to that as a stable interface.</p>



<a name="239491897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239491897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239491897">(May 19 2021 at 20:36)</a>:</h4>
<p>I personally agree with <span class="user-mention" data-user-id="239881">@Josh Triplett</span> that stabilizing this in Cargo is pretty far out, if ever, to happen. That being said in the meantime there may be another possible alternative. We don't really consider the stability of Cargo's own codebase that much. We very liberally change tons of things lots of the time (often in breaking ways). The stability guarantees of Cargo only really uphold through the CLI interface, which we can clearly demarcate and all that.</p>



<a name="239491989"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239491989" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joe Hendrix <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239491989">(May 19 2021 at 20:36)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> I appreciate your reflections and input.  I have some thoughts about how to manage the interface problem, but maybe I should first develop a proof of concept.</p>
<p>If it is publicly available, I'd appreciate pointers to the tool you are building.  For reference, the options I'm currently considering are:</p>
<ol>
<li>Work on improving the existing <code>leanpkg</code> system.</li>
<li>Fork <code>cargo</code> and lean-specific features as needed.</li>
<li>The Rust CXX approach of hooking into <code>build.rs</code>.</li>
<li>This "external command" approach described here.</li>
</ol>
<p>I mostly write Haskell professionally, and would prefer to find a solution to Lean package management that doesn't require yet another package manager.  For my background, I'm aware of the many challenges associated with building cross-language tools.   I've built binary analysis libraries that support multiple ISAs as well as symbolic execution/verification tools that support multiple languages.  I've also done a bit of proof-of-concept work showing how to call Lean from Rust and vice-versa.  It was pretty straightforward through C ABI functions.  </p>
<p>Right now, I'm still leaning towards approach 4 with the idea this would be a fork if there is no upstream interest, but I'm certainly open to other ideas -- perhaps another package manager would be more amenable to this  goal.</p>



<a name="239491998"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239491998" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239491998">(May 19 2021 at 20:36)</a>:</h4>
<p>This means that, if it works, we could consider refactoring Cargo to make it easier to plug in other languages. For example Cargo could be internally refactored so you could build your own copy of "cargo-lean" or something like that which would have lean support and/or rust integration</p>



<a name="239492194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239492194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239492194">(May 19 2021 at 20:38)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> Yeah, that's a fair point. Or more generally, you could create a <code>cargo-external</code> that provides <code>cargo external build</code> which could provide such integration.</p>



<a name="239492339"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239492339" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239492339">(May 19 2021 at 20:39)</a>:</h4>
<p>I'm saying this with the hope/suspicion that the Rust-specific bits of building code in Cargo are somewhat easily separable in a way that if things were refactored it wouldn't cause too much of a burden on us developing Cargo itself. That would then allow development of external integration which plugs in lean and/or other languages and such.</p>
<p>The main goal of this would be enabling <span class="user-mention" data-user-id="399826">@Joe Hendrix</span> to build on the work that Cargo has already done but without causing the Cargo team too many issues (e.g. large stable interfaces to discuss, huge code changes which otherwise are difficult to work with, etc)</p>



<a name="239493245"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239493245" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joe Hendrix <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239493245">(May 19 2021 at 20:47)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> Minimizing existing codebase churn is one of my goals.  I don't really want to introduce issues into Cargo itself, and would prefer a fairly narrow interface that made the expectation languages that want this adapt to cargo.  For example, the actual build step itself may look virtually identical to the existing call to <code>rustc</code> but with a different command invoked.  It'd be up to the language to make sure it supported a similar interface (or fail gracefully if a feature is not supported).</p>



<a name="239493350"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239493350" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239493350">(May 19 2021 at 20:48)</a>:</h4>
<p>indeed yeah! I would initially predict that such integration in Cargo would not be overly hard, but I don't know for certain</p>



<a name="239493385"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239493385" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239493385">(May 19 2021 at 20:48)</a>:</h4>
<p>I think, personally, this would be fine to land in Cargo itself. That way you could work on the lean integration externally and it wouldn't diverge all that much really from Cargo tiself</p>



<a name="239493413"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239493413" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239493413">(May 19 2021 at 20:49)</a>:</h4>
<p>in that I'm thinking there will still be a fork, just one that should be relatively easy to rebase</p>



<a name="239514761"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239514761" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joe Hendrix <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239514761">(May 20 2021 at 00:09)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> That approach sounds good to me.  Thank you for the feedback.</p>
<p>In terms of timing, I'm hoping to have a fork available for the Lean community to try out sometime this year.  If this <code>cargo-lean</code> fork does seem like it will meet the needs of the Lean community, then I'd start with an official RFC process to get feedback on the design.  In the meantime, I may propose modest changes to cargo that don't affect functionality but make it easier to maintain the fork.</p>



<a name="239598582"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/239598582" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#239598582">(May 20 2021 at 14:15)</a>:</h4>
<p>Ok!</p>



<a name="245464289"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/extern%20languages%3F/near/245464289" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joe Hendrix <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/extern.20languages.3F.html#245464289">(Jul 09 2021 at 16:22)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> As an update, I was able to patch cargo over a month ago so it can fork out to an external process for building, computing checksums, etc.  I think that's going to be fairly easy to maintain.</p>
<p>Since then some urgent work came up, but I hope to resume getting the actual Lean build integration in mid August.</p>
<p>I hoped to make more steady process, but still am planning to pursue this.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>